1. Field of the Invention
The present disclosure relates to an image processing device that performs image composition processing on multiple captured images and outputs a composite image, an imaging system and an imaging apparatus that include the image processing device, and an image processing method.
2. Description of the Related Art
A wide angle image that cannot be obtained with one camera can be generated in a seamless state, by performing so-called stitching processing that composes images that are captured by multiple cameras and generates one composite image (a panoramic image). In this stitching processing, two adjacent cameras are arranged in such a manner that portions of image capture areas overlap, and images are composed by superimposing image areas of boundary portions that correspond to overlapping portions of image capture areas of two cameras, on each other, or by suitably performing trimming.
On the other hand, when photographic subjects that are positioned at greatly different distances from the camera are present, that is, a photographic subject in the background and a photographic subject in the foreground are present in the overlapping portions of the image capture areas of the two cameras, a defect occurs in that a state where deviation takes place in a positional relationship between a shape of the photographic subject in the background and a shape of the photographic subject in the foreground, that is, so-called parallax occurs between captured images that are acquired by two cameras, and thus, in a composite image, the same two foreground shapes appear in an overlapping manner or one portion of the foreground shape is lost. In the stitching, parallax correction that suppresses a defect in the image due to the parallax is performed.
With regard to this parallax correction, for example, Japanese Patent Unexamined Publication No. 2010-50842 discloses that, with block matching that is based on an edge or a feature quantity, a positional relationship between shapes of photographic subjects that appear in captured images that are acquired by two cameras is acquired, and parallax correction that deforms an image is performed based on this information. Particularly, with this technology, a stitching point that defines the degree to which the image is deformed at the time of the parallax correction is changed for every frame, and a suitable composite image is generated for every frame.
Furthermore, for example, Japanese Patent No. 5225313 discloses that a curved stitching boundary is set for two images in such a manner that an image area in which a foreground shape such as that of a person appears, that is, an image area in which a cost function is high is avoided, trimming that cuts off the two images along the stitching boundary is performed, and then image composition is performed. Particularly, with this technology, optimization of the stitching boundary is set to be performed considering the continuity of frames, and, thus, the stitching boundary that is temporally stable can be set. Because of this, when a composite image is displayed as a moving image, an occurrence of an unnatural motion in the shape of the photographic subject that appears in the image can be avoided.
However, when the stitching point is changed for every frame, and a change in the image, which is likely to have an influence on the parallax between the two images, occurs, the stitching point changes greatly between frames. As a result, a sudden change in the image occurs between the frames, and when the composite image is displayed as a moving image, an unnatural motion occurs in the shape of the photographic subject that appears in the image. This gives a viewer a feeling that something is wrong.
On the other hand, if the stitching boundary that is temporally stable is set, when the composite image is displayed as a moving image, the occurrence of the unnatural motion in the shape of the photographic subject that appears in the image can be avoided. However, an amount of computation in the stitching processing is large, and in a case where information on a future frame is necessary, it is difficult to perform real time processing that, in real time, outputs the image which is captured in the camera while performing the stitching processing.
Additionally, when the stitching processing is performed on the captured images (the composed images) that are acquired by the multiple cameras in which imaging conditions are set individually, without any change, a boundary of an area of each of the captured images in the composite image is made to be unnaturally conspicuous by a difference in brightness or shade between the captured images.
It is also thought that the imaging conditions relating to exposure or white balance are set equally for all the cameras. However, a case can occur in which the exposure or the white balance is not suitably set in the camera that acquires the captured image (for example, a captured image that has high importance, which a user is likely to pay attention to) that is a reference, and in which an imaging target is not suitably displayed (for example, blown-out highlight, blocked-up shadows, or the like occur).